hmon: add heartbeat monitor by arkjedrz · Pull Request #67 · eclipse-score/lifecycle

arkjedrz · 2026-02-09T15:43:16Z

Add heartbeat monitor HMON.

Resolves #68

github-actions · 2026-02-09T15:44:17Z

License Check Results

🚀 The license check job ran with the Bazel command:

bazel run //:license-check

Status: ⚠️ Needs Review

Click to expand output

[License Check Output]
Extracting Bazel installation...
Starting local Bazel server (8.4.2) and connecting to it...
INFO: Invocation ID: 6e26fdd5-d5d7-4759-b2c3-342c6d26aa47
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
WARNING: For repository 'score_rust_policies', the root module requires module version score_rust_policies@0.0.3, but got score_rust_policies@0.0.5 in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
    currently loading: 
Loading: 0 packages loaded
    currently loading: 
Loading: 0 packages loaded
    currently loading: 
Analyzing: target //:license-check (1 packages loaded, 0 targets configured)
Analyzing: target //:license-check (1 packages loaded, 0 targets configured)

Analyzing: target //:license-check (54 packages loaded, 9 targets configured)

Analyzing: target //:license-check (99 packages loaded, 34 targets configured)

Analyzing: target //:license-check (153 packages loaded, 2667 targets configured)

Analyzing: target //:license-check (154 packages loaded, 4850 targets configured)

Analyzing: target //:license-check (160 packages loaded, 7852 targets configured)

Analyzing: target //:license-check (165 packages loaded, 7901 targets configured)

Analyzing: target //:license-check (165 packages loaded, 7901 targets configured)

Analyzing: target //:license-check (165 packages loaded, 7901 targets configured)

Analyzing: target //:license-check (168 packages loaded, 9789 targets configured)

Analyzing: target //:license-check (168 packages loaded, 9789 targets configured)

Analyzing: target //:license-check (169 packages loaded, 9913 targets configured)

Analyzing: target //:license-check (169 packages loaded, 9913 targets configured)

INFO: Analyzed target //:license-check (170 packages loaded, 10039 targets configured).
[12 / 16] JavaToolchainCompileClasses external/rules_java+/toolchains/platformclasspath_classes; 0s disk-cache, processwrapper-sandbox ... (2 actions running)
[14 / 16] JavaToolchainCompileBootClasspath external/rules_java+/toolchains/platformclasspath.jar; 0s disk-cache, processwrapper-sandbox
INFO: Found 1 target...
Target //:license.check.license_check up-to-date:
  bazel-bin/license.check.license_check
  bazel-bin/license.check.license_check.jar
INFO: Elapsed time: 27.606s, Critical Path: 2.59s
INFO: 16 processes: 12 internal, 3 processwrapper-sandbox, 1 worker.
INFO: Build completed successfully, 16 total actions
INFO: Running command line: bazel-bin/license.check.license_check ./formatted.txt <args omitted>
usage: org.eclipse.dash.licenses.cli.Main [-batch <int>] [-cd <url>]
       [-confidence <int>] [-ef <url>] [-excludeSources <sources>] [-help] [-lic
       <url>] [-project <shortname>] [-repo <url>] [-review] [-summary <file>]
       [-timeout <seconds>] [-token <token>]

github-actions · 2026-02-09T15:45:35Z

The created documentation from the pull request is available at: docu-html

Copilot

Pull request overview

Adds a new heartbeat monitor (HMON) to the Rust health monitoring library and integrates it into the existing monitoring worker/supervisor notification flow.

Changes:

Introduces heartbeat module (monitor + atomic state) and integrates heartbeat monitors into HealthMonitorBuilder/HealthMonitor.
Updates the monitor evaluation interface to accept a shared hmon_starting_point and wires it through the monitoring worker thread.
Refactors SupervisorAPIClient into a dedicated module with selectable implementations via Cargo features.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
src/health_monitoring_lib/rust/worker.rs	Passes a shared HMON start instant into monitor evaluations; moves supervisor client trait out.
src/health_monitoring_lib/rust/supervisor_api_client/mod.rs	Adds feature-selected `SupervisorAPIClient` + implementation alias.
src/health_monitoring_lib/rust/supervisor_api_client/stub_supervisor_api_client.rs	New stub client implementation.
src/health_monitoring_lib/rust/supervisor_api_client/score_supervisor_api_client.rs	New SCORE client implementation.
src/health_monitoring_lib/rust/lib.rs	Adds heartbeat monitors to builder + start flow; uses new supervisor client impl selector.
src/health_monitoring_lib/rust/heartbeat/mod.rs	Exposes heartbeat monitor API.
src/health_monitoring_lib/rust/heartbeat/heartbeat_state.rs	Adds atomic packed heartbeat state and tests.
src/health_monitoring_lib/rust/heartbeat/heartbeat_monitor.rs	Implements heartbeat monitor logic + tests (incl. loom).
src/health_monitoring_lib/rust/deadline/deadline_monitor.rs	Adapts deadline evaluation to new evaluator signature and shared start instant.
src/health_monitoring_lib/rust/common.rs	Extends evaluation error types; adds `duration_to_u32`; updates evaluator trait signature.
src/health_monitoring_lib/Cargo.toml	Adds optional `monitor_rs`, loom target dep, and feature defaults.
src/health_monitoring_lib/BUILD	Enables `score_supervisor_api_client` feature in Bazel builds.
Cargo.toml	Updates workspace defaults and adds `cfg(loom)` lint configuration.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/health_monitoring_lib/rust/heartbeat/heartbeat_monitor.rs

src/health_monitoring_lib/rust/deadline/deadline_monitor.rs

src/health_monitoring_lib/rust/heartbeat/heartbeat_monitor.rs

src/health_monitoring_lib/rust/supervisor_api_client/mod.rs

src/health_monitoring_lib/rust/supervisor_api_client/score_supervisor_api_client.rs

src/health_monitoring_lib/rust/supervisor_api_client/mod.rs

src/health_monitoring_lib/rust/heartbeat/heartbeat_monitor.rs

Copilot

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/health_monitoring_lib/Cargo.toml

src/health_monitoring_lib/rust/heartbeat/heartbeat_monitor.rs

Copilot

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/health_monitoring_lib/rust/common.rs

src/health_monitoring_lib/rust/supervisor_api_client/score_supervisor_api_client.rs

src/health_monitoring_lib/rust/supervisor_api_client/stub_supervisor_api_client.rs

Copilot

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/health_monitoring_lib/rust/supervisor_api_client/score_supervisor_api_client.rs

src/health_monitoring_lib/rust/heartbeat/heartbeat_monitor.rs

.github/workflows/lint_clippy.yml

Copilot

Pull request overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-02T14:20:54Z

src/health_monitoring_lib/rust/heartbeat/heartbeat_monitor.rs

+
+    /// Create range with values offset by timestamp.
+    fn offset(&self, timestamp: u32) -> Self {
+        Self::new(self.min + timestamp, self.max + timestamp)


InternalRange::offset adds timestamp to min/max with unchecked u32 arithmetic. This can wrap silently in release builds near the u32 limit, causing wrong range comparisons. Use checked_add (and handle overflow explicitly) rather than +.

Suggested change

Self::new(self.min + timestamp, self.max + timestamp)

let min = self

.min

.checked_add(timestamp)

.expect("overflow when computing min offset in InternalRange::offset");

let max = self

.max

.checked_add(timestamp)

.expect("overflow when computing max offset in InternalRange::offset");

Self::new(min, max)

Copilot · 2026-03-02T14:20:55Z

src/health_monitoring_lib/rust/heartbeat/heartbeat_state.rs

+        assert!(value < 1 << 29, "provided heartbeat offset is out of range");
+        self.0 = ((value as u64) << BEAT_OFFSET) | (self.0 & !BEAT_MASK);


set_heartbeat_timestamp_offset uses an unconditional assert! on the 29-bit limit. Since this can be reached from normal runtime behavior (e.g., long time between heartbeats), consider returning a Result/saturating instead of panicking, or ensure all callers validate inputs so this cannot trigger in production.

Suggested change

assert!(value < 1 << 29, "provided heartbeat offset is out of range");

self.0 = ((value as u64) << BEAT_OFFSET) | (self.0 & !BEAT_MASK);

let clamped = min(value, (1 << 29) - 1);

self.0 = ((clamped as u64) << BEAT_OFFSET) | (self.0 & !BEAT_MASK);

Copilot · 2026-03-02T14:20:55Z

src/health_monitoring_lib/rust/heartbeat/heartbeat_monitor.rs

+            let hmon_starting_point = Instant::now();
+
+            // Wait until in range.
+            sleep_until(Duration::from_millis(50), hmon_starting_point);
+


These loom tests rely on wall-clock time (Instant::now(), elapsed()) and real sleeping (sleep_until -> std::thread::sleep) inside loom::model. This is typically non-deterministic and can make the model exploration extremely slow/flaky. Prefer loom-friendly tests that avoid real time (e.g., drive state transitions directly or gate out time-based tests from the loom configuration).

Suggested change

let hmon_starting_point = Instant::now();

// Wait until in range.

sleep_until(Duration::from_millis(50), hmon_starting_point);

// Simulate that 50 ms have already elapsed since the monitor starting point.

let hmon_starting_point = Instant::now() - Duration::from_millis(50);

Copilot · 2026-03-02T14:20:55Z

src/health_monitoring_lib/rust/common.rs

+    #[should_panic(expected = "HMON starting point is earlier than monitor starting point")]
+    fn hmon_time_offset_wrong_order() {
+        let hmon_starting_point = Instant::now();
+        let monitor_starting_point = Instant::now();
+        let _offset = hmon_time_offset(hmon_starting_point, monitor_starting_point);


hmon_time_offset_wrong_order relies on two consecutive Instant::now() calls producing strictly increasing values. On platforms with coarse timer resolution the instants can be equal, so checked_duration_since returns Some(0) and the test won’t panic (flaky test). Make the ordering deterministic by deriving one instant from the other (e.g., monitor_starting_point = hmon_starting_point.checked_add(...) or vice versa).

src/health_monitoring_lib/rust/heartbeat/heartbeat_monitor.rs

Copilot · 2026-03-02T14:20:56Z

src/health_monitoring_lib/rust/heartbeat/heartbeat_monitor.rs

+        // Check range is valid.
+        let range_min_ms = self.range.min.as_millis() as u64;
+        let internal_processing_cycle_ms = internal_processing_cycle.as_millis() as u64;
+        if range_min_ms * 2 <= internal_processing_cycle_ms {
+            error!(
+                "Internal processing cycle duration ({} ms) must be longer than two shortest allowed ranges ({} ms).",
+                internal_processing_cycle_ms, range_min_ms
+            );
+            return Err(HealthMonitorError::InvalidArgument);
+        }


HeartbeatStateSnapshot can only store a 29-bit heartbeat offset (max ~6.2 days in ms) and will panic if a heartbeat arrives after that (set_heartbeat_timestamp_offset asserts). HeartbeatMonitorBuilder::build should validate that the configured range (and any expected max time between heartbeats) fits this representation, and return InvalidArgument instead of allowing a runtime panic later.

Copilot · 2026-03-02T14:20:56Z

src/health_monitoring_lib/rust/heartbeat/heartbeat_monitor.rs

+    fn evaluate(&self, hmon_starting_point: Instant, on_error: &mut dyn FnMut(&MonitorTag, MonitorEvaluationError)) {
+        // Get current timestamp, with offset to HMON time.
+        let offset = hmon_time_offset(hmon_starting_point, self.monitor_starting_point);
+        let now = offset + duration_to_u32(hmon_starting_point.elapsed());


now is computed as offset + duration_to_u32(hmon_starting_point.elapsed()) using u32 addition. In release builds this can wrap silently once elapsed > u32::MAX - offset (i.e., earlier than the 49-day limit if offset != 0), leading to incorrect evaluations. Use checked_add (and fail/return an error) to avoid wraparound.

Suggested change

let now = offset + duration_to_u32(hmon_starting_point.elapsed());

let elapsed = duration_to_u32(hmon_starting_point.elapsed());

let now = match offset.checked_add(elapsed) {

Some(value) => value,

None => {

error!("Overflow while computing current timestamp in HeartbeatMonitorInner::evaluate");

return;

}

};

pawelrutkaq · 2026-03-04T10:04:40Z

FYI: We are still dropping a refactor on this code today, as we discussed after review with @arkjedrz

Add heartbeat monitor HMON.

Rework heartbeat monitor state into two atomics.

arkjedrz requested a review from pawelrutkaq February 9, 2026 15:43

arkjedrz had a problem deploying to workflow-approval February 9, 2026 15:43 — with GitHub Actions Failure

arkjedrz requested a review from Copilot February 9, 2026 15:43

Copilot AI reviewed Feb 9, 2026

View reviewed changes

arkjedrz force-pushed the arkjedrz_heartbeat-monitor branch from f07c0db to e65f6a6 Compare February 10, 2026 11:43

arkjedrz temporarily deployed to workflow-approval February 10, 2026 11:43 — with GitHub Actions Inactive

arkjedrz requested a review from Copilot February 10, 2026 11:44

Copilot AI reviewed Feb 10, 2026

View reviewed changes

src/health_monitoring_lib/Cargo.toml Outdated Show resolved Hide resolved

src/health_monitoring_lib/rust/heartbeat/heartbeat_monitor.rs Outdated Show resolved Hide resolved

src/health_monitoring_lib/rust/heartbeat/heartbeat_monitor.rs Outdated Show resolved Hide resolved

arkjedrz force-pushed the arkjedrz_heartbeat-monitor branch from e65f6a6 to bd438c3 Compare February 10, 2026 12:57

arkjedrz temporarily deployed to workflow-approval February 10, 2026 12:57 — with GitHub Actions Inactive

arkjedrz force-pushed the arkjedrz_heartbeat-monitor branch from bd438c3 to c5f4b84 Compare February 10, 2026 13:14

arkjedrz had a problem deploying to workflow-approval February 10, 2026 13:14 — with GitHub Actions Failure

arkjedrz requested a review from Copilot February 10, 2026 13:15

Copilot AI reviewed Feb 10, 2026

View reviewed changes

arkjedrz force-pushed the arkjedrz_heartbeat-monitor branch from c5f4b84 to b012f37 Compare February 10, 2026 13:19

arkjedrz temporarily deployed to workflow-approval February 10, 2026 13:19 — with GitHub Actions Inactive

arkjedrz requested a review from Copilot February 10, 2026 13:19

Copilot AI reviewed Feb 10, 2026

View reviewed changes

arkjedrz force-pushed the arkjedrz_heartbeat-monitor branch from b012f37 to 862da21 Compare February 11, 2026 12:14

arkjedrz temporarily deployed to workflow-approval February 11, 2026 12:14 — with GitHub Actions Inactive

arkjedrz force-pushed the arkjedrz_heartbeat-monitor branch from 862da21 to 59c92ee Compare February 13, 2026 14:48

arkjedrz temporarily deployed to workflow-approval February 13, 2026 14:48 — with GitHub Actions Inactive

arkjedrz force-pushed the arkjedrz_heartbeat-monitor branch from 59c92ee to cf14efb Compare February 17, 2026 12:12

arkjedrz temporarily deployed to workflow-approval February 17, 2026 12:12 — with GitHub Actions Inactive

arkjedrz self-assigned this Feb 17, 2026

arkjedrz force-pushed the arkjedrz_heartbeat-monitor branch from cf14efb to 67fe6cc Compare February 25, 2026 09:30

arkjedrz had a problem deploying to workflow-approval March 2, 2026 12:21 — with GitHub Actions Failure

arkjedrz force-pushed the arkjedrz_heartbeat-monitor branch from 656c5fe to 0014124 Compare March 2, 2026 14:12

arkjedrz marked this pull request as ready for review March 2, 2026 14:13

arkjedrz had a problem deploying to workflow-approval March 2, 2026 14:13 — with GitHub Actions Failure

arkjedrz had a problem deploying to workflow-approval March 2, 2026 14:13 — with GitHub Actions Error

arkjedrz requested a review from Copilot March 2, 2026 14:13

Copilot started reviewing on behalf of arkjedrz March 2, 2026 14:13 View session

Copilot AI reviewed Mar 2, 2026

View reviewed changes

arkjedrz force-pushed the arkjedrz_heartbeat-monitor branch from 0014124 to a8b11fc Compare March 3, 2026 10:16

arkjedrz temporarily deployed to workflow-approval March 3, 2026 10:16 — with GitHub Actions Inactive

arkjedrz force-pushed the arkjedrz_heartbeat-monitor branch from a8b11fc to ef78463 Compare March 3, 2026 12:39

arkjedrz temporarily deployed to workflow-approval March 3, 2026 12:39 — with GitHub Actions Inactive

arkjedrz force-pushed the arkjedrz_heartbeat-monitor branch from ef78463 to 3816630 Compare March 4, 2026 09:35

arkjedrz requested a deployment to workflow-approval March 4, 2026 09:36 — with GitHub Actions Waiting

pawelrutkaq mentioned this pull request Mar 4, 2026

hmon: add C++ interface to heartbeat monitor #72

Open

arkjedrz force-pushed the arkjedrz_heartbeat-monitor branch from 3816630 to 27766ce Compare March 4, 2026 11:32

arkjedrz requested a deployment to workflow-approval March 4, 2026 11:33 — with GitHub Actions Waiting

arkjedrz force-pushed the arkjedrz_heartbeat-monitor branch from 27766ce to 77da130 Compare March 4, 2026 11:34

arkjedrz requested a deployment to workflow-approval March 4, 2026 11:34 — with GitHub Actions Waiting

arkjedrz added 2 commits March 4, 2026 14:53

hmon: add heartbeat monitor

c7f46aa

Add heartbeat monitor HMON.

hmon: rework heartbeat monitor state

125905c

Rework heartbeat monitor state into two atomics.

arkjedrz force-pushed the arkjedrz_heartbeat-monitor branch from 77da130 to 125905c Compare March 4, 2026 13:54

arkjedrz deployed to workflow-approval March 4, 2026 13:54 — with GitHub Actions Active

arkjedrz temporarily deployed to workflow-approval March 4, 2026 13:54 — with GitHub Actions Inactive

-        Self::new(self.min + timestamp, self.max + timestamp)
+        let min = self
+            .min
+            .checked_add(timestamp)
+            .expect("overflow when computing min offset in InternalRange::offset");
+        let max = self
+            .max
+            .checked_add(timestamp)
+            .expect("overflow when computing max offset in InternalRange::offset");
+        Self::new(min, max)

		assert!(value < 1 << 29, "provided heartbeat offset is out of range");
		self.0 = ((value as u64) << BEAT_OFFSET) \| (self.0 & !BEAT_MASK);

-        let now = offset + duration_to_u32(hmon_starting_point.elapsed());
+        let elapsed = duration_to_u32(hmon_starting_point.elapsed());
+        let now = match offset.checked_add(elapsed) {
+            Some(value) => value,
+            None => {
+                error!("Overflow while computing current timestamp in HeartbeatMonitorInner::evaluate");
+                return;
+            }
+        };

Conversation

arkjedrz commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

License Check Results

Uh oh!

github-actions bot commented Feb 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

pawelrutkaq commented Mar 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

arkjedrz commented Feb 9, 2026 •

edited

Loading

github-actions bot commented Feb 9, 2026 •

edited

Loading